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ABSTRACT 

Classroom teachers frequently express concern that 
textbooks and other curriculum materials are too difficult for their 
students. The objectives of this investigation were to develop a 
method for scoring the Level of Abstraction (LOA) of science reading 
material, and explore its relationship with certain other known 
methods for assessing curriculum materials (passage readability 
level, student cloze score, and teacher prediction of students* level 
of comprehension). Also examined were relationships between students' 
cloze scores and passage readability levels. The study was based upon 
nine passages taken from life, earth, and physical science textbooks 
written at three different levels. Non-significant correlations were 
found between: (1) passage LOA and passage readability; (2) student 
cloze scores and passage LOA; (3) passage LOA and teacher prediction 
of student success; and (4) student cloze scores and passage 
readability level. However, the consistent and high, but 
non-significant correlations between LOA and cloze scores combined 
with the fact that the LOA is based upon deep structure rather than 
surface structure of written material indicate the importance of 
further investigation of the LOA in its relationship to student 
comprehension of written material. (Author/TW) 
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Abstract 

Classroom teachers frequently express concern that tex tbooks 
and other curriculum materials are too difficult for their 
students. One criterion for textbook selection has been 
readability level; however, the grade level designation does not 
necessarily reflect the cognitive demand that concepts within the 
textbook passages place on students or how well they comprehend 
the mater ial . 

The objectives of this investigation were to: (a) develop a 
method for scoring the Level of Abstraction (LOA) of science 
reading material, and (b) explore its relationship with certain 
other known methods for assessing curriculum materials (passage 
readability level, student cloze score, and teacher prediction of 
students' level of comprehension). Also examined were 
relationships between students' cloze scores and passage 
readabil i ty levels . 

The study was based upon nine passages taken from life, 
earth, and physical science textbooks written at three different 
levels. Data were collected to obtain preliminary agreement from 
science educators in the classification of concepts, determine 
interscorer reliability, and compare student and passage 
variables. Participants consisted of 24 science educators, 60 
science teachers from elementary, middle, and high schools, and 
425 urban students in Grades 5, 7, and 10. Instruments used 
were: (a) the procedure to determine level of abstraction 
developed in this study, (b) the cloze procedure, and (c) 
"Reading Level Analysis: Programs for Teachers," a published 
computer program which calculates readability levels based upon 
several well-known methods. 

Level of Abstraction (LOA) of printed material is defined as 
the ratio of postulat ional or not concrete concepts to the total 
number of concepts in a written passage expressed as a percent, 
and is conceptually independent of readability level. 

Non-significant correlations were found between: a) passage 
LOA and passage readability level, b) student cloze scores and 
passage LOA, c) passage LOA and teacher prediction of student 
success, and d) student cloze scores and passage readability 
level. However, the consistent and high, but non-significant 
correlations between LOA and cloze scores combined with the fact 
that the LOA is based upon deep structure rather than surface 
structure of written material indicate the importance of further 
investigation of the LOA in its relationship to student 
comprehension of written material. 
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Purpose of the Study 

Classroom teachers frequently express concern that textbooks 
and other curriculum materials are too difficult for their 
students. Researchers have found that a large number of 
adolescents and adults are not functioning at the formal 
operational level (e.g., Lawson & Renner, 1975; Karplus, 1979; 
Shayer & Adey, 1981), yet curi-iculum materials contain much that is 
written at that level. Efforts to provide a better match are being 
made (Karplus, et al., 1979; Brooks, Fusco & Grennon, 1983; Shayer 
& Adey, 1981; Fumo, 1984) . Important for this purpose is the 
determination of the level of cognitive demand of curriculum 
materials. One criterion for textbook selection has been 
readability level; however, the grade level designation, which is 
generally based on word and sentence length, does not necessarily 
reflect the cognitive demand that concepts within the textbook 
passages place on students or how well students comprehend the 
mater ial . 

The objectives of this study were to: (a) develop a method for 
scoring the level of abstraction (LOA) of science reading material, 
and (b) explore its relationship with certain other known methods 
for assessing curr iculum mater ials including readability formulas 
(Jerman & Kropf, 1982), the cloze procedure (Herber, 1978, p. 237), 
and teacher prediction of student success in compreheiiding written 
passages. Also examined were relationships between students* cloze 
scores and passage readability levels. 



Definitions 

Level of abstraction (LOA) is defined in this study as the 
ratio of the number of postulat ional/ not concrete concepts (NC) 
to the total number of concepts (C + NC) in a passage from 
printed materials expressed as a percent (LOA = NC/ (C + tiC) X 
100) . Three terms used throughout the description of the 
procedure to determine LOA are concept, descriptive (or concrete) 
concept, and postulat ional (or not concrete) concept . Concept' 
refers to an idea of an object, action, or event. According to 
Klausmeir, Ghatala, and Frayer (1974), "concepts are fundamental 
agents of thought for human beings from early childhood through 
adult" (p. 1). Descriptive or concrete concepts are concepts 
whose referents can be experienced directly by scientists or 
other competent observers. One's understanding of this type of 
concept is based on direct experience with phenomena or the use 
of instruments that extend the range of the senses (e.g., 

^ telescopes) . Examples include table, rock, and comet . Benjamin 

(1955, p. 56) related this type of concept to the listing and 
describing of data. Postulat ional or not concrete concepts are 
concepts whose referents cannot be experienced directly by 
anyone. They must be understood in terms of other concepts, 
functional relationships, inferences, and/ or idealized models. 
Examples include atom, density, light-year, and plate tectonics. 
^ Benjamin (1955) stated that these "concepts involve the use of 
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methods of construction, inference, and insight or creative 
imagination" (p. 56) . 

Readability level of a passage is a determination of the 
level of difficulty of text material usually stated in terms of 
grade level (Herber, 1978, p. 16). This value is determined 
independent of the reader and takes into consideration the 
surface features of the passage, such as word and sentence 
length . 

The cloze procedure involves omitting words from a passage 
at regular intervals and requires the reader to determine from 
the context of the remaining words what these omitted words are. 
Success at this task has been shown to be a measure of a reader's 
comprehension of the material (De Santi, 1986). This score takes 
into consideration the deep structure of the passage as well as 
context clues. A low score indicates that the passage is 
difficult for the leader to comprehend. 

Teacher judgement of a student's level of success 
interacting with text material is a subjective observation that a 
student can read and understand text material. This opinion is 
based upon knowledge about b ':h the student and the written 
passages . 

Thus, variables explored in this study were: (a) student 
cloze scores, (b) passage readability levels, (c) passage level 
of abstraction (LOA) scores, and (d) teacher prediction of 
student success on the cloze tests. 



Hypotheses 

The following hypotheses were tested at a £<.05 level of 
s igni f icance . 

1. There is no significant correlation between mean 
readability levels and LOA scores for passages from 
life, earth, and physical science textbooks. 

2. There is no significant correlation between mean cloze 
scores of students and LOA scores for passages from 
life, earth, and physical science textbooks. 

3. There is no significant correlation between LOA scores 
of the passages and students' level of success in 
comprehending the three passages as predicted by their 
teachers. 

4. There is no significant correlation between cloze scores 
of students and readability levels of passages from 
life, earth, and physical science textbooks. 
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Procedures 



Nine samples of reading material were identified. These 
passages were selected from life, earth, and physical science 
textbooks intended for students in Grades 5, 7, and 10. These 
correspond with the grade levels of students who participated in 
the cloze testing* For each grade, one passage was selected from 
a life science textbook, one from an earth science textbook, and 
one from a physical science textbook. Each passage was 
approximately two hundred words in length and was selected from 
books not currently being used by the students participating in 
the study. Data were collected by determining the readability 
levels of these sample textbook passages, determining LOA scores 
for the passages, estimating students' level of success in 
comprehending the passages, and administering cloze test versions 
to groups of students. 

The computer program, "Reading Level Analysis: Programs for 
Teachers" (Jerman & Kropf, 1982), which calculates readability 
levels based upon the Fog Index, Dale-Chall Index, Fry Graph, and 
Flesch Grade Level was used to determine the readability level of 
passages analyzed' in this study. Mean readability levels were 
then calculated. 

Following an extensive review of the literature (Vachon, 
1987), the concept of LOA and the procedures for scoring passages 
were developed. See Appendix, These procedures were validated 
by having a group of science educators examine the method for 
identifying the concrete (descriptive) and not concrete 
(postulat ional) concepts in samples of textbook passages. The 
percentage of agreement among the 24 educators who classified the 
concepts was determined. 

For the purpose of determining the LOA, concept was defined 
as an object, event, or action (e.g., rock, photosynthesis, 
measuring) . These ideas are communicated in print by a noun or 
verb and its respective modifier(s). In classifying concepts, 
the context in which the concepts are written was carefully 
considered. Classification of concepts as either concrete or not 
concrete was based on their operational definitions. 

Using responses from the science educators, tentative LOA 
scores (TLOA) were determined for each of the passages by 
identifying concepts in each of the 200-word sample passages, 
classifying the concepts as either concrete or not concrete, and 
then determining the ratio of the number of not concrete concepts 
to the total number of concepts in the sample passages times 100. 

Samples of science teachers were also asked to score the 
passages using written directions only. A total of 60 teachers 
responded. The consistency among LOA scores obtained by teachers 
was examined by comparing the standard deviation of scores for 
each passage. In addition, Cronbach's alpha (Crocker & Algina, 
1986, p. 138; SPSS-X, 1983) was used to determine interscorer 
consistency among teachers scoring passages for each science area 
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(life, earth, and physical) . Because of the variability in 
scoring among teachers, detailed analysis of the responses 
provided by the teachers for each passage was completed to 
identify the sources of variation and also to obtain a composite 
LOA score (CLOA) based upon transformations made to cast out the 
effects of obvious scoring errors. 

Cloze passages were administered to students at Grade Levels 
5, 7, and 10 in nine public schools in a medium-sized urban 
school district. Descriptions of student population samples are 
presented in Table 1. 

Table 1 



Description of 


Classes in Which 


Cloze Tests Were Administered 


Descr iptor 




Grade 


level 




5 


7 


10 


All 


n 


85 


176 


164 


425 


Age range 


10-13 


12-14 


14-18 


10-18 


Age mean 


11.2 


12.9 


15.8 


13.7 


% male 


45 


53 


48 


49 


% female 


55 


47 


52 


51 


% black 


58 


74 


43 


51 


% non-black 


42 


26 


57 


49 



The 11 classroom teachers followed set directions which 
instructed them to: (a) record the names of their students; (b) 
read the three selected passages without deletions; (c) predict 
how well they thought each of their students would comprehend 
each passage and record a 3, 2, or 1 indicating high, medium, or 
low success, respectively; (d) read to their students the 
directions for filling in the 5th-word deletion cloze passages; 
(e) administer the practice cloze test one day; and, then (f) 
administer the cloze test consisting of three passages the next 
day. Materials were then returned for scoring. The exact word 
replacement method of scoring (Jongsma, 1980, p. 18) was used; 
however, misspelled words were accepted. The percent correct was 
computed and recorded. 

The hypotheses were tested using both TLOA and CLOA scores 
by applying Pearson's Product Moment Correlational analyses which 
specified pairwise deletion of missing data and 2-tailed tests of 
significance. 



Results 

Preliminary validation of the LOA produced a high level of 
agreement ranging from 87% to 92% among science educators who 
were asked to respond to the dichotomous classification of 
^ concepts in three of the sample textbook passages. 
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Standard deviations found when analyzing the teacher- 
determined LOA scores for each of the nine passages varied widely* 
Cronbach's alpha coefficients ranged from .24 to .15, also 
indicating varied reliability among teachers using only written 
directions to score the different passages* 

Pearson's Product Moment Correlation Coefficients between 
pairs of variables are shown in the following tables: (a) LOA 
scores of the science passages and their readability levels - Table 
2/ (b) LOA scores of the passages and students* mean cloze scores - 
Table 3/ (c) LOA scores of the passages and students' mean levels 
of success predicted by their teachers - Table A, and (d) mean 
readability levels of passages and students' roean performance on 
cloze tests - Table 5. The negative correlations between pairs of 
variables in Tables 3, A, and 5 were expected because of the 
reversed scales of the variables being compared* TLOA scores 
(determined using feedback provided by science er'ucators) and CLOA 
scores (determined by casting out obvious scoring errors of 
teachers) were both used in the analyses* 

As shown in Table 2, correlations between LOA scores and 
readability levels within grade levels tended to be medium-low, 
except for the Grade 7 passages* Across grade levels, the passages 
are not ranked in the same order by either the TLOA or CLOA scores; 
and, they are ranked in a third way by the readability scores* 
This leads one to suspect that readability and LOA are measures of 
different characteristics* 

Table 2 

Correlations Between LOA Scores and Readability Levels of 
Science Passages 

Readabil i ty 



Grade level TLOA CLOA 



5 


(n=3) 
Life 


5.0 


20 






26 








Earth 


6.2 


17 


r = 


.29 


34 


r = 


.44 




Phys ical 


6.3 


27 






26 






7 


(n=3) 
Life 


11.0 


52 






41 








Earth 


5 . 3 


25 


r = 


.94 


18 


r= 1 


.00 




Phys ical 


6.6 


21 






25 






10 


(n = 3) 
Life 


9.1 


20 






20 








Earth 


7.9 


37 


r = 


.39 


34 


r = 


.35 




Phys ical 


10.4 


47 






41 






All 


(n=9) 




r = 


.80 


* * 


r = 


.65 





*p<.05. **p<.01. 
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Overall, consistency among teachers using only written 
directions varied and the coefficients based upon CLOA scores were 
not as high as those in correlational analyses using TLOA scores. 

When the relationship between student cloz'j scores and LOA 
scores was examined, it was found that correlations using TLOA were 
medium-high to high, while only the correlation using CLOA scores 
for Grade 10 was high. See Table 3. The high correlations could 
indicate that the LOA raeasures an important factor which 
contributes to a student's comprehension of printed materials. 
Because the cloze test measures student interaction with the 
reading materials and involves student characteristics such as 
motivation and interest, significant correlations may not be 
reasonably expected. 

Table 3 

Correlations Between LOA Scores of Science Passages and 
Students ' Mean Cloze Scor.es 



Grade Cloze TLOA CLOA 



5 


(n=3) 
















Li f e 


35.0 


20 






26 






Earth 


31.9 


17 


r = 


-.87 


34 


r= .30 




Physical 


21.5 


27 






26 




7 


(n=3) 
















Li f e 


31.3 


52 






41 






Earth 


33.4 


25 


r = 


-.77 


18 


r= -.44 




Physical 


40.1 


21 






25 




10 


(n = 3) 
















Life 


51.0 


20 






20 






Earth 


44.3 


37 


r = 


-.93 


34 


r= -.92 




Physip^l 


29.2 


47 






41 





*p<.05. **p<.01. 



The high correlations between mean levels of success predicted 
by teachers and TLOA scores at Grades 5 and 7, and CLOA scores at 
Grade 7 may indicate that the teachers recognized the abstractness 
of the passages and considered this characteristic when making 
their predictions of students' success in comprehending the 
material. See Table 4. 
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Table 4 



Correlations Between LOA Scores of Science Passages and 
Students' Mean Levels of Success Predicted by Their Teachers 



Grade 



Level of 
Success 



TLOA 



CLOA 



10 



(n = 3) 

Li f e 

Earth 

Physical 

(n = 3) 

Life 

Earth 

Physical 

(n=3) 

Life 

Earth 

Physical 



1.9 
1.8 
1.7 

1.6 
1.7 
1.7 

1.7 
1.9 
1.6 



20 
17 
27 



25 
21 



r= - .85 



r= - .93 



26 

34 r = 
26 



- .26 



20 

37 r 
47 



= - .12 



41 
18 
25 

20 
34 
41 



r= -1.00** 



r= - .07 



*p<.05. **p<.01. 

When the relationship between readability of the passages and 
cloze scores of students based on these passages was examined, two 
of the three correlations ware found to be medium-high (r::-.65 and 
£=-.73 at Grades 5 and 10, respectively), and the third much lower 
(£=-.51). See Tf.ble 5. Because the cloze scores do not correlate 
as highly with readability as they do with LOA scores, one could 
suspect that the characer istics used in calculating readability do 
not influence reader comprehension as much as the level of 
abstraction at which a passage is written. 

Table 5 

Correlations Between Mean Readability Levels of Science 
Passages and Students' Mean Performance on Cloze Tests 



Grade 



Readabil ity 
level 



Cloze score 



10 



(n=3) 






Life 


5.0 


35.0 


Earth 


6.2 


31 .9 


Physical 


6.3 


21.5 


(n=3) 






Life 


11.0 


31.3 


Earth 


5.3 


33.4 


Physical 


6.6 


40.1 


(n=3) 






Life 


9.1 


51.0 


Earth 


7.9 


44.3 


Physical 


10.4 


29 .2 



*p<.05. **p<.01. 
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r= -.73 



r= -.51 



r= -.65 
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Interpretations 

The cloze inventory measures the reader's: (a) readirig 
comprehension, (b) logical language production, (c) ability to deal 
with the grammatical structures of printed language, (d) word 
recognition, and (e) word identification (De Santi, 1986, p. 1). 
This was borne out in this study. Further reinforcement for this 
interpretation was found in studying the relationships between the 
cloze scores and teacher predictions of student success in 
comprehending written passages, and the students' reading levels 
and the cloze test scores (Vachon, 1987). It can be assumed that 
teochers make their predictions based upon their knov;ledge about a 
student's reading ability and other factors such as motivation. 

The LOA seems to be related to these five student abilities 
measured by the cloze proce-.ure. As a result of this study, one is 
led to suspect that the LOA, which is based on the deep structure 
of the written passage, is tapping a factor which contributes to 
student comprehension of written materials. 

Training and practice appear to be important factors in 
improving interscorer reliability among teachers who follow the 
procedure for determining the LOA. 



Conclusions 

Because written curriculum materials, particularly the 
textbook, are basic tools in most science classrooms, students must 
be provided with materials that take into consideration the 
cognitive level of the reader for whom they were written. 
Classroom teachers are in a position to know the characteristics of 
their students and can judge how well their students will 
comprehend selected passages particularly when the teacher 
recognizes the cognitive demand of the written material. Knowledce 
about particular students may not be available to individuals 
responsible for selecting textbook materials; therefore, methods 
for assessing the characteristics of written material are 
important . 

Readability formulas are widely used by educators to determine 
the appropriateness of textbooks and other curriculum materials for 
students at different grade levels. However, because these formulas 
are based on word and sentence length, the grade level designation 
can be easily manipulated without altering the conceptual demand the 
reading material places on the reader. Readability formulas focus 
on the surface structure of printed materials and do not necessarily 
present accurate estimates of a student's ability to comprehend a 
v/ritten passage. Results from cloze tests provide a more accurate 
assessment of the cognitive demand of written material on students, 
but the method requires time to test groups of students and scoring 
several passages. On the other hand, LOA scores are independent of 
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any student performance^ but still appear to Le closely related to 
the actual performance of students on cloze tests. 

Further study will determine the usefulness of the LOA 
procedure for assessing science reading materials and providing 
guidelines for authors writing curriculum materials for our 
students . 
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Append ix : 

Procedure for Determining the Level 
of Abstraction of Science Read ing Mater ial 



1. Read a passage 200 words in length and underline each noun or 
verb phrase that stands for a concept^ but omit all 
structural words^ such as: articles (e.g.^ a, an, the); 
conjunctions (e.g., and, but, or, nor, because); prepositions 
(e.g., in, at, for, on, with); and forms of the linking verb 
"to be" (e.g., be, am, are, was), 

a. Note : When a concept extends to the next line, draw an 
arrow to indicate the connection (e.g., series circ ui t in 
the example in part c) . 

b . Spec ial Situations ; 

1) Chemical or mathematical symbols and formulas - count 
as words. 

2) Chemical or mathematical equations - count as 
sentences . 

3) Idioms (e.g., kick the bucket yvhich means die) - 
count as a single unit. 

4) Gaps/ deletions (e.g., doesn't have holes in "Pumice 
has holes, but granite doesn't.") insert the missing 
word(s) and include the concept in the count. 

5) Fillers/ insertions (e.g.. There in "There are three 
boys going skating.", inasmuch , consequently ) - do 
not count. 

6) "Of" phrases which can be transformed into noun 
phrases (e.g., height of a tree meaning tree's 
height) - count as a single unit. 

7) Pronouns - count except in relative clauses (e.g., 
who in "The man who came late was my father."). 

c. Example : Becky's lamp is not wired with a series — > 
circuit . If _it were * , all the light bulbs would be 
unl it > In a series circuit , the current cannot pass 
through a burned-out bulb to reach the others . The 
light bulbs in Becky's lamp are connected in a 
parallel circuit . In a parallel circuit , there is 
more than one path for the current to take . 

*meaning were wired 
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2. Reread the passage carefully. 

a. Consider the context in which the concepts are written, 
and classify each concept as either C for concrete or NC 
for not concrete using the following definitions: 

1) concrete (C) means the concept has referents which 
can be experienced directly by scientists and other 
competent observers . They are understood based on 
direct experience with phenomena or the use of 
instruments that extend the range of the senses 
(e.g., telescopes) - leave under 1 ined . (Examples: 
table, chair, rock, moon, water, look.) Do not try 
to j udge whether or not the intended reader can 

ex per ience the concept by direct observation. 

2) not concrete (NC) means the concept has referents 
which canno t be exper ienced directly by anyone . They 
must be understood in terms of other concepts, 
functional relationships, inferences, and/ or 
idealized models - under 1 ine again . (Examples: 
atom, densi ty , mole^ ideal gas, TTght-year , plate 
tectonics) . 

b . Important : 

1) Context - Consider the meaning within the context of 
the passage because some concepts can be either C or 
NC. "Temperature" is an example. When "temperature" 
'ts" used to refer to the hotness or coldness of an 
object, it is C. When temperature is used in terms 
of the speed at which molecules move, it is NC . 

2) Analogues and other models - These may aid iir^concept 
understanding, but themselves are usually not 
concrete even though they may be used to clarify an 
NC concept (e.g., reference to rocks as radioactive 
UTocks) . 

3) Processes or actions - Even though the result can be 
experienced, the process or action itself may still 
be NC. (For example, we can see a bulb light up 
because of the electrical current running through the 
wires, but the current of electrons is NC.) 

4) Pronouns or references made to pr ev ious~concepts - 
Look at the antecedent (word or phrase) . If any part 
of the antecedent is NC, mark the pronoun NIC (see 
number 5 in EXAMPLES Be'low) . 

5) Historical names and dates - Consider as NC. 
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c. EXAMPLES: some rocks found in the earth are 

4 5 6 

over a mill ion years old . They formed when the 



7 8 9 

temperature of the earth cooled . Rad ioact ive clocks 



10 11 
in rocks began ticking . 



1 Some rocks - can be directly observed. 

2 found - an action within experience. 

3 ear th - can be directly observed. 

4 million years old - length of time beyond anyone's 
exper lence . 

5 They - pronoun referring to "some rocks" which can be 
directly observed. 

6 formed - even though it refers to an action to which 
students can relate; in this case, forming 

a rock is a process which cannot be directly 
observed . 

7 temperature of the earth - meaning earth's 
temperature can be directly observed or measured. 

8 cooled - an action within experience. 

9 Radioactive clocks - an analogue to aid students, but 
tnese " ciocKs" cannot be directly observed. 

10 rocks - can be directly observed. 

11 began t icki ng - an analogue to aid students, but is 
not an action within experience. 

3. Count the total number of concepts (C + NC) . Count the 
number of NC concepts. 

4. Use this formula to determine the level of abstraction: 

LOA = NC X 100 

C + NC 
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